A Case Study of Participant Feedback for a Teaching Remotely Professional Learning Course
This case study serves as the final project for the ECI 588 Text Mining in Education course offered in spring 2021 at North Carolina State University. This course is part of the Learning Analytics Certificate Program.
The Rmarkdown file can be downloaded from the code folding control button in the upper right hand of this page. The full R Studio project can be access via this Github repository. However, it should be noted that the data for this project is not included because it requires special permission for sharing.
Context of this Case Study
The data source for my final project consists of the end of course survey responses for the “Teaching Remotely: A Practical Guide” 2020 MOOC-Ed course offered by the Friday Institute for Educational Innovation. This professional learning course was offered to help educators address the key issues of transitioning to Emergency Remote Learning due to the COVID-19 global pandemic. The course objectives covered key professional development areas for teaching remotely such as establishing norms for remote learning, bring social-emotional learning to the virtual classroom, maintaining connections with student and families, digital content selection, supporting learners with special needs, and providing quality feedback. You can read more about the course on the Friday Institute’s Professional Learning and Collaboration Environment course site.
The raw data includes 3,080 end of course survey responses where participants provided their ratings for overall effectiveness of the course, their perceived effectiveness of particular aspects of the course, their estimated number of hours spent engaged in the course activities, and their ratings for how effectively the course improved their practice. The main focus of my project for this course will be the open-ended, unstructured text responses that participants provided in answer to the following questions:
What was the most valuable aspect of this course in supporting your personal or professional learning goals?
In what ways, if any, do you anticipate applying the knowledge, skills, and/or resources you acquired from this course to your professional practice?
Please describe any changes you have made to your practice, including how you have applied the knowledge, skills, and/or resources you gained in this course.
What recommendations do you have for making this course more valuable to future participants (e.g., other resources, additional features, activities, etc.)? Please explain.
Guiding Questions
I am interested in this data/context because I am heavily involved in developing educator professional development and this analysis may help guide such development. In particular, my guiding questions for this case study will be:
What aspects of the Teaching Remotely professional development course do participants find most valuable?
What changes, if any, are being made to participants’ practice as a result of their participation in this course?
If participants found the course to be ineffective, what specific recommendations have they made for its improvement?
How do the sentiments of participants’ open-ended responses compare to their ratings of the course’s overall effectiveness?
Target Audience
My target audience is any other researcher or teacher educator who, like myself, is interested in making improvements to their own professional development offerings. I hope that the analysis will help identify “best practices” by examining the components of professional learning that are most valuable as well as specific areas that could be targeted for improvement.
Limitations of this Case Study
Before diving into the analysis of this data, it is important to note that responses to the end of course survey were voluntary. Therefore, it is possible that those participants who did not complete all of the activities or that found the course to be ineffective did not respond to the end of course survey. Examining the data does show that the ratings for overall effectiveness of the course is skewed very positively which confirms that a key limitation for this case study lies in the response rates for those participants who only completed a portion of the course or were dissatisfied with the course content in some manner.
End of Course Survey Analysis
In the following sections, the processes used to analyze the end of course survey data are described and findings are presented.
Data Wrangling
My initial pre-processing of the data began with removing some HTML formatting that appears in text responses and setting column names that are more readable than those assigned by the survey system. I chose not to omit NA values at this point simply because there was at least one in every response and that resulted in no data left to analyze! Instead, I omit the NA values at appropriate points throughout my analysis. Additionally, since some cursory exploration of the data showed that participants made mention of specific unit numbers, I also chose not to remove numbers/digits from the text responses when I pre-processed the data. The following shows the clean column names and a glimpse of the data contained in the end of course survey responses.
eoc_survey_data <- read_csv("data/eoc_surveydata_wrangled_grouped.csv") %>%
select(response,
EffectivenessRating,
IntroDiscussionRating,
CourseContentRating,
PuttingItTogetherRating,
ActionPlanRating,
ResourcesRating,
MostValuableAspects,
ImprovementEstablishingNorms,
ImprovementBringingSEL,
ImprovementMaintainingConnection,
ImprovementSelectDigResources,
ImprovementSupportSpecialPop,
ImprovementProvidingFeedback,
PositiveChangesEffectiveness,
HasAttemptedChanges,
AnticipatedApplicationPractice,
ChangesToPractice,
CourseRecommendations,
DesiredActivityCompletion,
CourseHoursEstimate,
text) %>%
mutate(CourseHoursEstimate = as.factor(CourseHoursEstimate)) %>%
mutate(CourseCompletion = as.factor(ifelse(DesiredActivityCompletion == 0, "incomplete", "complete"))) %>%
filter(response != 87684,
response != 89417)glimpse(eoc_survey_data)## Rows: 3,078
## Columns: 23
## $ response <dbl> 80338, 87032, 89153, 79595, 92739, 81…
## $ EffectivenessRating <dbl> 1, 1, 2, 2, 2, 2, 2, 2, 1, 2, 1, 1, 2…
## $ IntroDiscussionRating <dbl> 1, 1, 2, 2, 2, 3, 2, 3, 3, 2, 1, 2, 2…
## $ CourseContentRating <dbl> 1, 1, 2, 2, 2, 2, 2, 2, 1, 2, 1, 1, 2…
## $ PuttingItTogetherRating <dbl> 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 1, 2, 2…
## $ ActionPlanRating <dbl> 1, 2, 2, 2, 3, 2, 2, 2, 2, 3, 1, 2, 3…
## $ ResourcesRating <dbl> 1, 1, 2, 1, 2, 1, 2, 2, 1, 1, 1, 1, 2…
## $ MostValuableAspects <chr> "I loved all of the attached resource…
## $ ImprovementEstablishingNorms <dbl> 1, 1, 2, 2, 2, 2, 2, 2, 1, 2, 1, 2, 1…
## $ ImprovementBringingSEL <dbl> 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2…
## $ ImprovementMaintainingConnection <dbl> 1, 1, 2, 1, 2, 2, 2, 3, 1, 2, 1, 2, 2…
## $ ImprovementSelectDigResources <dbl> 1, 2, 2, 3, 3, 2, 2, 2, 2, 2, 1, 2, 2…
## $ ImprovementSupportSpecialPop <dbl> 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1…
## $ ImprovementProvidingFeedback <dbl> 1, 1, 2, 2, 2, 2, 2, 2, 1, 3, 1, 1, 1…
## $ PositiveChangesEffectiveness <dbl> 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 1, 1, 1…
## $ HasAttemptedChanges <dbl> 1, 1, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3…
## $ AnticipatedApplicationPractice <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ ChangesToPractice <chr> "I really like the ideas with the stu…
## $ CourseRecommendations <chr> "I think that I made recommendations …
## $ DesiredActivityCompletion <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1…
## $ CourseHoursEstimate <fct> 1 : 1-2 hours per unit, 2 : 3-4 hours…
## $ text <chr> "i loved all of the attached resource…
## $ CourseCompletion <fct> complete, complete, complete, complet…As you will notice, two of the survey responses were discarded due to corrupted character encodings within the text responses. I believe this may have been due to participants copying and pasting material into their responses. The following sections present my analysis of the Teaching Remotely MOOC-ED end of course survey responses arranged by the guiding questions that frame the case study.
Most Valuable Aspects
The first guiding question for my analysis is:
What aspects of the Teaching Remotely professional development course do participants find most valuable?
Based on the nature of the responses, which are pretty short, the text was tokenized and stop words were removed. Since the responses contained some HTML from the survey system, custom stop words were also employed for the tokenization process (e.g., “nbsp” or “amp”). It should be noted that some of the MOOC-Ed participants chose not to respond to the survey question: “What was the most valuable aspect of this course in supporting your personal or professional learning goals?” Therefore, I had to do some further pre-processing at this stage and omit any NA values that were lurking in the data.
eoc_aspects <- eoc_survey_data %>%
select(response, MostValuableAspects) %>%
slice(-1, -2) %>%
na.omit()
eoc_aspects %>%
select(MostValuableAspects)## # A tibble: 3,076 x 1
## MostValuableAspects
## <chr>
## 1 Learning about teaching remotely and how it affects teachers and students.&n…
## 2 To see that the effort of the last few months and my long-term engagement ha…
## 3 Discussion responses
## 4 I am going to rethink my way of working.
## 5 Validation of what my district and myself are doing as remote teaching conti…
## 6 I think the resources that were provided in the course were most helpful for…
## 7 It is obviously very timely! I like having a structure to start applyi…
## 8 I liked seeing examples to get ideas to use with my students.
## 9 The numerous resources and strategies provided
## 10 I really enjoyed the videos from all of the units. The links and resou…
## # … with 3,066 more rowsCustom stop words were employed for the leftover HTML that crept into the responses (e.g., nbsp or amp) and for words that were simply a restating of the survey prompt (e.g., valuable and aspect) as part of the response.
custom_stop <- data.frame("word" = c("nbsp",
"amp",
"NA",
"N/A",
"N/a",
"Na",
"na",
"aspect",
"valuable"))
eoc_tidy_aspects <- eoc_aspects %>%
unnest_tokens(word, MostValuableAspects) %>%
anti_join(stop_words) %>%
anti_join(custom_stop)Below you can see the words that appear in the question responses at least 150 times. Resources is at the top of the list, but ignoring some words like helped or helpful there are others that stand out such as ideas, links, videos, articles, feedback, and discussion.
eoc_tidy_aspects %>%
dplyr::count(word, sort = TRUE) %>%
filter(n > 150) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(n, word)) +
theme_minimal() +
geom_col(fill="#7B0D1E") +
labs(y = NULL)
It was at this point that I decided to examine bigrams for this question. As you can see in the bar chart, some of the most frequently used words are plan and unit. These often appear in the responses as part of phrases, such as “action plan” or “unit 3” and I hoped to capture more specific valuable aspects of the MOOC-Ed such as these if at all possible.
eoc_aspects_bigrams <- eoc_aspects %>%
unnest_ngrams(bigram, MostValuableAspects, n = 2, to_lower = TRUE)
eoc_separated_aspects <- eoc_aspects_bigrams %>%
separate(bigram, into = c("word1", "word2"), sep = " ") %>%
na.omit()
eoc_united_aspects <- eoc_separated_aspects %>%
filter(!(word1 %in% stop_words$word),
!(word2 %in% stop_words$word)) %>%
filter(!(word1 %in% custom_stop$word),
!(word2 %in% custom_stop$word)) %>%
unite(bigram, c(word1, word2), sep = " ")
bigram_counts_aspects <- eoc_united_aspects %>%
dplyr::count(bigram, sort = TRUE) %>%
top_n(25)As you can see from the word cloud below, examining the bigrams was actually fruitful. It revealed that aspects like “action plan”, “learning goals”, “unit 5”, and “unit 6” were mentioned quite frequently by those who responded to the end of course survey. It also revealed other key content like “social emotional”, “special education”, “establishing norms” and “digital citizenship”.
wordcloud2(bigram_counts_aspects, size = 2, shape='circle', color='#7B0D1E')Changes to Practice
The second guiding question for my analysis is:
What changes, if any, are being made to participants’ practice as a result of their participation in this course?
Answering this question focuses in on the following two survey questions:
- In what ways, if any, do you anticipate applying the knowledge, skills, and/or resources you acquired from this course to your professional practice?
- Please describe any changes you have made to your practice, including how you have applied the knowledge, skills, and/or resources you gained in this course.
Responses to these questions were captured in the dataframe columns called AnticipatedApplicationPractice and ChangesToPractice. Let’s first focus on the changes that participants anticipated making to their practice. We’ll just grab the AnticipatedApplicationPractice column from our dataframe and do some further pre-processing by stripping out lingering HTML tags and responses where the participants just wrote some variant of “N/A.”
anticipated_changes <- eoc_survey_data %>%
select(response, AnticipatedApplicationPractice) %>%
mutate(AnticipatedApplicationPractice = gsub(' ', ' ', AnticipatedApplicationPractice)) %>%
mutate(AnticipatedApplicationPractice = gsub('&', ' ', AnticipatedApplicationPractice)) %>%
slice(-1, -2) %>%
na.omit() %>%
filter(AnticipatedApplicationPractice != 'N/A')
anticipated_changes %>%
select(AnticipatedApplicationPractice)## # A tibble: 78 x 1
## AnticipatedApplicationPractice
## <chr>
## 1 "I have made copies of several of the articles. I will continue to read the…
## 2 "I plan to incorporate knowledge into my tutoring sessions."
## 3 "I plan to share the ideas provided through professional development with ad…
## 4 "I would like to apply the SEL info maintaining relationships."
## 5 "I plan on being more cautious with the online materials that I send my stud…
## 6 "I've selected the resources I think most apply to my school community and w…
## 7 "I know that I will immediately apply many of the skills such as defined nor…
## 8 "We plan to use many of the resources made available."
## 9 "I plan on usinsome of the strategies that I read about in the units."
## 10 "The knowledge I learned from this will be applied in my professional practi…
## # … with 68 more rowsThere are only 78 responses to this survey question about anticipated changes to participants’ professional practice, so we’ll just grab a sampling of those to examine them for any patterns. In particular, we will target the responses that specifically mention a participant’s “plan” and grab ten of those to read more closely.
changes_quotes <- anticipated_changes %>%
select(AnticipatedApplicationPractice) %>%
filter(grepl('plan', AnticipatedApplicationPractice))
sample_changes <- sample_n(changes_quotes, 10)kable(sample_changes)| AnticipatedApplicationPractice |
|---|
| I wanted to answer the last question honestly because I have not started to actually plan for this year. I have bookmarked many of the resources and will print out and laminate them for constant reference. I will use videos to introduce myself and record lessons. I will incorporate digital citizenship in my lessons and create a very structured and routine remote learning platform for my class. I have also bookmarked the parent/student surveys which I will use frequently to elicit feedback. |
| I plan on being more cautious with the online materials that I send my students. I also plan on focusing a lot on SEL and maintaining connections with my students. |
| the reason why I clicked “no” was that at the time of doing this course our school system was on vacation so there would have been no way to implement that which was learnt. That being said, There is clear intent on my part to implement content learnt. How do I plan on doing this? With the online class beginning in earnest from September 1st 2020, The notes that I have gathered will be used to create the environment of learning for my class, the mode of feedback and assessment and the atmosphere for learning and achievement for both students and parents. |
| I plan on usinsome of the strategies that I read about in the units. |
| I plan to look at some of the new resources that I learned about that may be better then ones that I am presently using. I hope to find the easiest and more effective ones to use with my students. |
| I plan to focus my attention on starting the year with plenty of relationship Building activities online (since we are beginning remotely), being sure to check in with my students and their parents on a regular basis to ensure how they are feeling, Proving links to the content I include and giving credit to the owner of it when necessary. This will also allow students to access it if needed. Being more alert to the needs of my disabled students To ensure they are able to continue learning. I will also be more consistent and specific in my feedback. |
| I plan to establish norms differently than I would in the classroom when I meet my kids next week through a Google Meet. I will be using some of the resources such as ck-12 when I am teaching. I will remember the importance of parent communication during remote learning and be more aware of what I can do to help parents in order to help their students. |
| I plan to utilize the techniques and resources from this course for the fall. Further, I hope to encourage and teach others in my department how to use these resources. |
| I plan on using a contract to clearly lay out expectations for all parties involved. |
| I plan to use many of the resources provided in my classroom, especially with assessments and reaching special needs students. I plan to start my course off with discussion digital citizenship which is something I usually don’t formally do. |
Even with just this handful of responses, we can see that participants plan to: 1) establish norms and provide social-emotional learning (SEL) through relationship-building activities in their virtual classrooms; 2) maintain effective communication with students and their families; 3) and share the content they learned with colleagues. Since some of these are learning objectives for the MOOC-Ed Teaching Remotely course, these responses appear to indicate that participants can envision how the course content aligns with and how they can apply it to their own professional practice.
Now that we have seen how participants anticipate applying the course content, let’s examine the changes they indicate they have already made to their practice. In this case, we’ll concentrate on the ChangesToPractice column of our dataframe.
practice_changes <- eoc_survey_data %>%
select(response, ChangesToPractice) %>%
mutate(ChangesToPractice = gsub(' ', ' ', ChangesToPractice)) %>%
mutate(ChangesToPractice = gsub('&', ' ', ChangesToPractice)) %>%
slice(-1, -2) %>%
na.omit() %>%
filter(ChangesToPractice != 'N/A')
practice_changes %>%
select(ChangesToPractice)## # A tibble: 2,565 x 1
## ChangesToPractice
## <chr>
## 1 "While I'm not actually teaching at the moment, I have implemented those ste…
## 2 "I consider the online resources I choose more carefully. "
## 3 "I will have to think about assessment and special needs in my teaching pr…
## 4 "I will focus on setting technology use norms with my students and families.…
## 5 "My 7th grade team is in the process of creating our norms before school sta…
## 6 "Just getting started, but great ideas to apply"
## 7 "I have cut back on the workload and found ways to help struggling learners.…
## 8 "I have included in my action plan key strategies to work on starting with c…
## 9 "I loved the last two units and am planning on focusing a lot on making sure…
## 10 "I have begun to redesign my learning materials ensuring that I utilize the …
## # … with 2,555 more rowsLooking at the tokenized responses to this survey question reveals that participants will focus changes to their practice on students, resources, feedback, norms, and SEL. They also frequently mention tools, planning, assessment, and communication.
eoc_tidy_changes <- practice_changes %>%
unnest_tokens(word, ChangesToPractice) %>%
anti_join(stop_words)
eoc_tidy_changes %>%
dplyr::count(word, sort = TRUE) %>%
filter(n > 150) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(n, word)) +
theme_minimal() +
geom_col(fill="#7B0D1E") +
labs(y = NULL)
Since these unigrams only convey where participants will focus their changes to practice, I decided to also examine bigrams here to get a clearer picture of the changes participants have made to their classroom practices.
eoc_changes_bigrams <- practice_changes %>%
unnest_ngrams(bigram, ChangesToPractice, n = 2, to_lower = TRUE)
eoc_separated_changes <- eoc_changes_bigrams %>%
separate(bigram, into = c("word1", "word2"), sep = " ") %>%
na.omit()
eoc_united_changes <- eoc_separated_changes %>%
filter(!(word1 %in% stop_words$word),
!(word2 %in% stop_words$word)) %>%
unite(bigram, c(word1, word2), sep = " ")
token_counts_changes <- eoc_tidy_changes %>%
dplyr::count(word, sort = TRUE)
bigram_counts_changes <- eoc_united_changes %>%
dplyr::count(bigram, sort = TRUE) %>%
top_n(25)wordcloud2(bigram_counts_changes, size = 2, shape = 'circle', color = '#7B0D1E')
The top 25 bigrams shown in the word cloud above suggest that participants have already made changes to their professional practice that include establishing norms, holding office hours, making an action plan or lesson plans, check ins, providing feedback, and using Google Classroom. However, many of the bigrams imply the same changes with different wordings, such as “providing feedback” and “provide feedback” or “virtual learning” and “online learning.” This suggests that future analyses may benefit from stemming.
Recommendations for Improvement
The next guiding question for this case study deals with participants’ recommendations for course improvements:
If participants found the course to be ineffective, what specific recommendations have they made for its improvement?
Specifically, it focuses on the responses from participants who rated the overall effectiveness of the course as Neither effective nor ineffective, ineffective, or very ineffective. The response text comes from the CourseRecommendations column of the dataframe.
participant_recommendations <- eoc_survey_data %>%
select(response, EffectivenessRating, CourseRecommendations) %>%
mutate(CourseRecommendations = gsub(' ', ' ', CourseRecommendations)) %>%
mutate(CourseRecommendations = gsub('&', ' ', CourseRecommendations)) %>%
slice(-1, -2) %>%
na.omit() %>%
filter(CourseRecommendations != 'N/A',
CourseRecommendations != 'n/a',
EffectivenessRating >= 3)
participant_recommendations %>%
select(EffectivenessRating, CourseRecommendations)## # A tibble: 69 x 2
## EffectivenessRating CourseRecommendations
## <dbl> <chr>
## 1 3 "Including more online student activities people have fo…
## 2 3 "Because I was rushed through it, make sure future parti…
## 3 4 "Somehow I would like a little more interaction with oth…
## 4 3 "Sharable contents and folders with activities in google…
## 5 3 "I don't have any."
## 6 3 "With remote learning being new it is hard to even provi…
## 7 5 "I think the course was one of the best I have taken."
## 8 5 "I thought unit 3 needed more tips for finding online op…
## 9 5 "The copyright module got a little technical for my unde…
## 10 5 "In some topics you have a lot of links and a lot of res…
## # … with 59 more rowsrecs_stop <- data.frame("word" = c("n/a", "N/A", "N/a"))
eoc_tidy_recs <- participant_recommendations %>%
unnest_tokens(word, CourseRecommendations) %>%
anti_join(stop_words) %>%
anti_join(recs_stop)
eoc_tidy_recs %>%
dplyr::count(word, sort = TRUE) %>%
filter(n > 2) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(n, word)) +
theme_minimal() +
geom_col(fill="#7B0D1E") +
labs(y = NULL) 
After tokenization, the words most frequently used in response to the survey question “What recommendations do you have for making this course more valuable to future participants?” are words like videos, time, teachers, resources. In fact, on the whole the tokenized text closely resembles the results from examining the other text responses. Therefore, a closer look into the raw data may be helpful. We’ll grab a sampling of the responses from the ratings of Neither effective nor ineffective, ineffective, or very ineffective and read through them.
recs_quotes <- participant_recommendations %>%
select(response, EffectivenessRating, CourseRecommendations) %>%
filter(EffectivenessRating >= 3,
CourseRecommendations != 'none',
CourseRecommendations != 'N/a')
sample_neutral <- recs_quotes %>%
filter(EffectivenessRating == 3) %>%
sample_n(5) %>%
select(CourseRecommendations)
sample_ineffective <- recs_quotes %>%
filter(EffectivenessRating == 4) %>%
select(CourseRecommendations)
sample_veryineffective <- recs_quotes %>%
filter(EffectivenessRating == 5) %>%
sample_n(5) %>%
select(CourseRecommendations)
sample_recs <- rbind(sample_neutral, sample_ineffective, sample_veryineffective)
kable(sample_recs)| CourseRecommendations |
|---|
| Please acknowledge that there is a population of students with disabilities that just cannot attend to remote learning. These families are being left behind and their teachers are feeling completely lost and overwhelmed. I get it, there are amazing resources for remote learning out there and I have spent much of my work days so far taking multiple required courses and trainings learning about them but I cannot use them. |
| I don’t have any. |
| Too long. |
| Shorting Unit 6 |
| This course is very redundant. There is a lot of information repeated in different ways, but it is still the same information and it’s not difficult information to grasp. Also, I felt like the course just bombarded me with educator blog posts, again on repetitive topics, and they were just anecdotal posts that created lists of stuff people should do. I think focusing on fewer, more research based resources would be more effective. |
| Somehow I would like a little more interaction with other participants. The posting was not as effective because it seemed there was not enough authentic engagement. One suggestion would be to have optional online, live discussions to further the conversations. |
| More information on children with Disabilities! |
| More interaction – zooms? |
| I found the course to be very valuable! I don’t have any suggestions. |
| I have no recommendations. I would keep it as is. |
| Maybe more practical examples on how some those best practices work, especially for lower level students. |
| In some topics you have a lot of links and a lot of resources to read. It becomes overwhelming. Pick two or three. |
| I think the course was one of the best I have taken. |
Reading through these results, some key issues are highlighted in these responses. For example, multiple participants in this sample suggests a lack of course content addressing remote teaching strategies for supporting students with disabilities. Other responses suggest that the course needed more interaction between participants or that course content was too long and/or repetitive. However, and perhaps somewhat surprisingly, some responses suggest that the course is sufficient and very valuable despite low overall effectiveness ratings. Thus, the next step in this case study will be to perform sentiment analysis on the participants’ text responses and compare that to the overall effectiveness ratings that participants gave to the course.
Participants’ Sentiments Compared to Course Ratings
The final guiding question for this study is:
How do the sentiments of participants’ open-ended responses compare to their ratings of the course’s overall effectiveness?
To answer this question, the response text of all three open-ended survey items was combined into a single text entry for each survey response. Additionally, participants’ ratings for the course’s overall effectiveness were adapted from the 5-point Likert scale (very effective to very ineffective) to a 3-point sentiment scale. That is, ratings of very effective and effective were collapsed to represent positive sentiment, neither effective nor ineffective represents a neutral sentiment, and very ineffective and ineffective were collapsed to represent a negative sentiment. Then proportions of the resulting course rating sentiments were calculated to make comparison to the results of sentiment analysis using the Valence Aware Dictionary and sEntiment Reasoner (VADER) package.
courseRatings_results <- eoc_survey_data %>%
mutate(sentiment = ifelse(EffectivenessRating == 5, "negative",
ifelse(EffectivenessRating == 4, "negative",
ifelse(EffectivenessRating == 3, "neutral",
ifelse(EffectivenessRating == 2, "positive", "positive"))))) %>%
dplyr::count(sentiment, sort = TRUE) %>%
na.omit() %>%
mutate(proportion = n / sum(n)) %>%
mutate(lexicon = "Participant Rating Sentiment")
courseRatings_results## # A tibble: 3 x 4
## sentiment n proportion lexicon
## <chr> <int> <dbl> <chr>
## 1 positive 2847 0.968 Participant Rating Sentiment
## 2 neutral 77 0.0262 Participant Rating Sentiment
## 3 negative 18 0.00612 Participant Rating Sentimentsurvey_resp_text <- eoc_survey_data
summary_vader <- vader::vader_df(survey_resp_text$text) %>%
select(text, compound, pos, neg, neu)
vader_results <- summary_vader %>%
mutate(sentiment = ifelse(compound > 0, "positive",
ifelse(compound < 0, "negative", "neutral"))) %>%
dplyr::count(sentiment, sort = TRUE) %>%
na.omit() %>%
mutate(proportion = n / sum(n)) %>%
mutate(lexicon = "Vader Sentiment")
vader_results## sentiment n proportion lexicon
## 1 positive 2648 0.86029890 Vader Sentiment
## 2 neutral 359 0.11663418 Vader Sentiment
## 3 negative 71 0.02306693 Vader SentimentAs you can see from the results below, the sentiment analysis of the open-ended responses appears to align with the sentiments of participants’ course ratings. This is despite some results above where participants rated the course to be ineffective or neither effective nor ineffective but stated that they liked the course as if and would not recommend changes.
comparison_df <- rbind(vader_results, courseRatings_results)
ggplot(comparison_df, aes(x = sentiment, y = proportion, fill = sentiment)) +
geom_col() +
facet_grid(. ~ lexicon) +
theme_minimal() +
theme(legend.position="none") +
theme(plot.title = element_text(hjust = 0.5)) +
xlab("Sentiment") +
ylab("Proportion") +
ggtitle("Sentiment Comparison of Participants' Ratings and Vader Analysis Results")
For a deeper look at participants’ sentiments as expressed through their text responses, I also decided to use the NRC sentiment lexicon to examine the range of emotions expressed across the survey responses. The results of this analysis are shown below.
nrc <- get_sentiments("nrc")
survey_stop <- data.frame("word" = c("nbsp", "NA", "N/A", "n/a", "na", "amp"))
participant_tokens <- eoc_survey_data %>%
select(response, text) %>%
unnest_tokens(word, text) %>%
anti_join(stop_words) %>%
anti_join(survey_stop)
sentiment_nrc <- inner_join(participant_tokens, nrc, by = "word")
summary_nrc <- sentiment_nrc %>%
dplyr::count(sentiment, sort = TRUE) %>%
spread(sentiment, n) %>%
mutate(lexicon = "nrc") %>%
relocate(lexicon) %>%
select(anger, anticipation, disgust, fear, joy, sadness, surprise, trust)
kable(summary_nrc)| anger | anticipation | disgust | fear | joy | sadness | surprise | trust |
|---|---|---|---|---|---|---|---|
| 395 | 4653 | 176 | 794 | 6229 | 643 | 1003 | 10278 |
As you can see, participants’ most frequently exhibited trust in their responses along with anticipation and joy. The least frequent emotions exhibited were anger and disgust. Again, these results support that, of those participants who responded to the end of course survey, overall satisfaction with the course content was skewed very positively.
Since the course ratings and sentiments are skewed so positively and a stated limitation of this data is a potentially low response rate for those who did not complete the course activities, we’ll examine the proportion of responses that indicate participants’ ability to complete all the activities they wanted to complete during the course. The survey item that provides the data for this part of the analysis is the yes or no question:
Were you able to complete all of the activities that you wanted to complete in this course?
Note here that we are considering a yes response to mean “complete” and a no response to mean “incomplete.”
courseCompletion_results <- eoc_survey_data %>%
select(CourseCompletion) %>%
dplyr::count(CourseCompletion, sort = TRUE) %>%
mutate(proportion = n / sum(n))
kable(courseCompletion_results)| CourseCompletion | n | proportion |
|---|---|---|
| complete | 2958 | 0.9610136 |
| incomplete | 120 | 0.0389864 |
As you can see in the table above, approximately 4 percent of the responses were from participants who were unable to complete all of the course activities as they desired. This may explain why the sentiment results are skewed so positively.
Discussion
Despite the short amount of text provided by the participants’ responses to the open-ended questions on the end of course surveys, there are still key insights to be gained that will aid refinements to our professional learning MOOC-Ed course on Teaching Remotely. In particular, some highlights from the analysis are:
- Participants found content such as action plans, learning goals, unit 5, and unit 6 to be particularly valuable aspects of the course.
- Participants expressed clear plans for applying course content to their own professional practice and are already doing so with practices such as establishing norms for their virtual classrooms, holding office hours and check ins, providing feedback, and using Google Classroom.
- Course content may benefit from identifying redundant material in an effort to shorten time spent, including more participant peer interaction, or adding content that addressing remote teaching supports/resources for those with students who have special needs, such as students with disabilities.
- On the whole, course ratings were very positive and the sentiments expressed by participants written responses agreed with these ratings.
Despite gaining these very useful insights from the end of course survey responses, we should consider that some insights may be missing due to the low response rates for those who were unable to complete the course. Finally, the results suggest that further text mining analysis may benefit from stemming the tokenized data.
The Author
Jennifer Houchins is a doctoral candidate in Learning Design and Technology at North Carolina State University. Her research examines students’ use of computational thinking and the effective use of instructional technology to deepen conceptual understanding in both formal and informal K-12 learning environments. She currently a graduate research assistant for the InfuseCS project, the Programmed Robotics in the School Makerspace (PRISM) project, and the Friday Institute’s Program Evaluation and Education Research (PEER) Group.
You can learn more about Jennifer and her work by visiting her website. You can also follow her on [@TooSweetGeek](https://twitter.com/TooSweetGeek).
Acknowledgements
This project uses techniques from Text Mining with R: A Tidy Approach by Julia Silge and David Robinson. Special thanks to our course instructor, Dr. Shaun Kellogg, for the sense of community created for this course as we all worked to learn R text mining techniques together.
End of Semester Feels
Bonus Content: This is Jenn. She is happy to be done with this crazy stressful year.
